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Abstract 

This  paper  describes  an  experimental  study  conducted  in  partnership  with  the  UCLA  School  of 
Engineering  and  Applied  Science  and  designed  to  demonstrate  and  evaluate  in  a  real-world  situation  the 
use  of  the  new  Automated  Team  Composition  System  (ATCS)  being  developed  by  Perceptronics 
Solutions  under  a  US  Army  Research  Institute  (ARI)  Small  Business  Innovative  Research  (SBIR)  project. 
The  ATCS  is  an  innovative  PC-based  tool  that  supports  rapid  team  formation  in  critical  incident 
preparedness  and  response  as  well  as  in  other  applications.  The  ATCS  uses  advanced  multiagent 
system  software  technology  to  help  command  and/or  decision  making  groups  solve  the  daunting 
organizational  problem  of  optimally  assigning  personnel  to  mission-oriented  teams.  The  team 
performance  criteria  used  by  the  Multiagent  System  derive  from  the  analysis  of  specific  mission  features 
as  well  as  from  the  psychological  literature.  The  present  study  used  archival  data  from  a  UCLA  (University 
of  California  Los  Angeles)  engineering  course  in  which  nominally  student  teams  prepare  a  research 
report  on  a  societal  problem  involving  both  technical  and  ethical  issues.  Data  on  approximately  700 
students  and  130  teams  were  first  examined  to  determine  the  relationships  among  student  characteristics 
(including  major,  gender,  ethnicity  and  essay  and  test  scores)  and  team  performance  (as  defined  by 
scores  on  several  components  of  the  research  report).  Analysis  of  the  data  revealed  a  number  of 
significant,  marginally  significant  and  trend  relationships  between  team  member  characteristics  and  team 
performance.  These  results,  in  combination  with  other  information  about  course  goals,  were  used  to 
program  the  ATCS  to  form  optimal  5-person  teams  with  specific  role  assignments.  The  ATCS  was  able 
within  1  minute  to  recommend  teams  from  the  candidate  population  that  met  all  the  programmed  criteria 
and  had  strong  face  validity  as  well  -  a  task  well  beyond  human  capabilities.  The  study  showed  that  the 
ACTS  can  be  readily  used  in  a  real-world  situation  in  which  rapid  team  formation  is  required  and  where 
knowledge  about  the  team  mission  can  be  combined  with  knowledge  about  the  factors  influencing  team 
performance.  Previous  ATCS  demonstrations  have  used  on-the-spot  questionnaires  to  gather  attribute 
and  qualification  data  about  team  candidates  in  anti-terrorist  exercises.  In  the  present  evaluation  we  used 
candidate  data  from  a  stored  data  base,  which  we  see  as  a  major  advantage  in  future  applications  of  the 
ATCS  tools,  and  also  used  a  non -military,  non-security  situation  to  demonstrate  the  broad  applicability  of 
the  technology. 

1.  Introduction 

Military  and  non-military  organizations  alike  need  optimally-constituted  teams  to  carry  out  immediate 
missions  and  long-term  objectives.  Of  particular  concern  are  missions  and  operations  requiring  the 
participation  of  diverse  services,  of  different  agencies  and  organizations,  and  possibly  also  of  partners 
dispersed  in  different  locations  [1]  [2], 

Bureaucratic  hierarchies  generally  do  not  solve  the  problem  of  connecting  people  with  complementary 
skills  and/or  mutual  interests,  and  consequently  these  structures  often  force  organizations  to  perform 
inefficiently  and  ineffectively  on  problems  that  require  rapid  and  team  composition  -  such  as  response  to 
civil  emergencies  or  critical  corporate  tasks  [3].  Often,  organizations  have  personnel  with  the  right  skill 
sets  to  meet  a  particular  challenge  or  to  create  innovative  solutions  to  difficult  problems,  but  are  unable  to 
carry  out  the  task  of  efficiently  and  effectively  putting  together  the  optimal  mix  of  personnel.  Previous 
research  has  shown  that  properly  designed  computer  support  tools  can  improve  human  ability  to  deal 
with  such  complex  organizational  functions.  The  goal  of  the  Automated  Team  Composition  project  is  to 
provide  such  a  team  formation  tool. 
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The  ATCS  team  formation  process  makes  certain  fundamental  assumptions  about  the  ingredients  of  an 
optimal  team  based  on  the  most  recent  organizational  research  findings  [4]  [5]  [6]  [7];  these  findings 
suggest  that  an  optimal  team  may  include: 

•  Qualified  Personnel.  Appropriate  position  and  technical  experience  and/or  knowledge. 

•  Good  Teamwork  Attitudes  and  Skills.  Motivation  and  know-how  that  can  support  effective 
cooperation  and  coordination. 

•  Social  Connections.  Team  members  are  centrally  connected  within  organizational  networks  and 
teams  have  high  within-team  connectivity. 

•  Appropriate  Weighting.  Relative  importance  weights  assigned  to  the  various  team  formation 
factors  -  for  each  team  role  and  for  the  team  as  a  whole. 

•  Team  Formation  Constraints.  Real-world  requirements  and  preferences  that  influence  both 
role  assignments  and  overall  team  formation. 

Optimizing  over  all  of  these  conditions  simultaneously  and  rapidly  for  a  reasonable  number  of  team  roles 
to  be  filled  from  a  sizable  pool  of  candidates  can  involve  choosing  from  among  millions  of  possible  team 
configurations.  Optimizing  this  task  rapidly  is  beyond  human  capabilities,  and  accordingly  the  job  can 
benefit  significantly  from  sophisticated  computational  processing.  The  ATCS  computational  framework 
described  below  provides  the  needed  rapid  processing  capability  by  applying  advanced  algorithms  in  a 
form  usable  by  non-experts  and  supportable  on  standard  PCs. 

2.  ATCS  Description 

Figure  1  shows  the  Automated  Team  Composition  System  developed  under  the  present  project  and  a 
related  DARPA  STTR  project  [8,  9].  The  main  components  are  described  briefly  in  the  following. 


_ 


Team  Composition  & 
Performance  Feedback 
Used  to  Adjust  System 
Parameters 


Figure  1  Automated  Team  Composition  System  (ATCS) 

•  Operational  Environment  is  composed  of  the  Organization,  which  can  be  military  or  non¬ 
military,  and  the  Mission,  which  is  the  identified  near-terms  or  long-term  objectives  and/or  goals  of 
the  Organization 
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•  Mission  Model  is  a  representation  of  the  mission  in  which  the  key  factors  are  identified  and 
expressed  in  terms  of  their  relative  importance  to  the  mission  as  a  whole 

•  Candidate  Roster  is  the  list  of  personnel  from  which  the  team  will  be  formed 

•  Performance  Model  is  a  representation  of  the  key  factors  characterizing  the  candidates;  these 
may  include: 

o  Qualifications,  such  as  organization,  rank,  experience  and  technical  knowledge,  based  on 
self  assessment  or  actual  performance  data; 

o  Teamwork  Measures,  which  are  teamwork  attitudes  and  skills  also  obtained  through 
questionnaire  or  available  data; 

o  Network  Measures,  these  are  social  networking  measures  -  such  as  individual  centrality 
within  a  social  network  and  team  connectivity  of  its  individual  members  -  which  again  can  be 
obtained  from  questionnaire  concerning  person-to-person  connections  or  directly  from 
communications  data  such  as  the  number  of  messages  exchanged  among  candidates. 

•  Multiagent  System  (MAS)  is  a  computational  system  consisting  of  multiple  software  agents 
where  each  agent  represents  a  candidate  along  with  his  or  her  attributes  and  capabilities,  and  the 
agents  “negotiate”  among  themselves  by  means  of  an  agent  interaction  protocol  to  solve  the 
distributed  constraint  optimization  problem  (DCOP)  and  select  the  members  of  the  locally  optimal 
team  or  teams  [1 0].  Critical  to  this  process  are: 

o  Optimization  Criteria,  which  are  the  rules  governing  how  the  various  attributes,  qualification 
and  social  networking  measures  are  combined  to  calculate  the  overall  criterion  by  which  the 
value  of  potential  teams  will  be  judged  as  well  as  the  size  of  the  “neighborhood”  over  which 
the  search  for  optimal  teams  will  be  conducted. 

o  Team  Formation  Constraints,  which  are  the  rules  pertaining  to  specific  team  roles  or  to  the 
overall  team  makeup,  and  are  respectively  defined  as  “single-agent”  or  “multi-agent” 
constraints 

•  Optimal  Teams  are  the  teams  selected  by  the  system  from  among  the  many  possible 
combinations,  taking  into  account  the  constraints  and  optimization  criteria.  The  selected  teams 
are  presented  to  the  users  as  recommendations,  and  may  be  modified  directly  or  by  changing  the 
original  constraints  and  optimization  criteria.  The  optimization  process  can  be  adaptively 
improved  over  time  by  observing  the  performance  of  the  selected  teams  and  using  that  feedback 
to  adjust  the  parameters 

The  ATCS  provides  a  human-factored  graphical  user  interface  that  makes  its  complex  support  tools  easily 
accessible  for  non-specialized  users.  For  example,  familiar  drag  and  drop  functions  allow  users  to  select 
rapidly  from  stored  templates  of  team  roles  and  performance  factors,  and  simple  screens  allow  users  to 
apply  factor  weights  to  the  roles  and  entire  team,  as  well  as  select  various  degrees  of  optimization  by 
computation  time  or  search  parameters. 

A  key  requirement  for  a  system  designed  to  support  critical  incident  response  is  compatibility  with  the 
National  Incident  Management  System  (NIMS).  NIMS  is  “a  consistent  approach  for  federal,  state,  tribal, 
and  local  governments  to  work  effectively  and  efficiently  together  to  prepare  for  and  respond  to  all 
hazards  including  acts  of  terrorism”  (National  Response  Plan).  ATCS  is  compatible  with  NIMS  and 
supports  its  major  functions. 

Overall,  ATCS  is  oriented  to  the  three  main  stages  in  team  composition,  which  are  applicable  to  general 
organizational  requirements  as  well  as  to  critical  incident  preparedness  and  response: 

•  Preplanning  Stage,  where  organizational  data  are  gathered,  the  several  models  are  created  and 
virtual  teams  are  configured  in  anticipation  of  an  event;  this  stage  may  also  highlight 
organizational  deficiencies; 

•  Response  Stage,  where  actual  teams  are  rapidly  formed  from  available  personnel  just  prior  to  a 
planned  for  event  or  after  an  unexpected  critical  incident  has  occurred;  and 
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•  Post-Event  Stage,  where  evaluation  of  actual  team  performance  is  used  as  an  aid  to 
organizational  improvement. 

The  ATCS  was  previously  successfully  demonstrated  during  anti-terrorist  exercises  conducted  by  the 
Center  for  Asymmetric  Warfare,  a  part  of  the  Naval  Postgraduate  School.  In  those  exercises  we  used  on- 
the-spot  questionnaires  to  gather  attribute  and  qualification  data  about  the  team  candidates;  and  in  those 
demonstrations  the  main  objective  was  to  show  that  the  ATCS  could  form  teams  in  the  required  time 
frame.  In  the  present  evaluation  we  used  candidate  data  from  a  stored  data  base,  which  we  see  as  a 
major  advantage  in  future  applications  of  the  ATCS,  and  also  used  a  non-military,  non-security  situation 
to  demonstrate  the  broad  applicability  of  the  technology. 

2.  Study  Design 

Target  Course  and  Team  Task.  The  UCLA  School  of  Engineering  and  Applied  Science  (SEAS)  Engr 
1 83  course  fulfils  both  the  School’s  ethics  and  writing  requirements,  and  according  is  taken  by  about  1 50 
students  each  of  the  four  quarters  it  is  offered.  The  course  requires  the  students  to  form  into 
multidisciplinary  teams  and  perform  a  typical  engineering  research  project  culminating  in  a  Team 
Research  Report.  Each  quarter  the  students,  with  some  assistance  from  the  instructors  and  TAs,  form 
themselves  into  about  25  to  27  three-  to  five-  person  teams.  Each  team  produces  its  Team  Research 
Report  on  a  topic  selected  by  the  team  from  a  suggested  list  of  five  or  six  topics  covering  current  societal 
problems  that  have  both  clear  technical  and  ethical  components.  The  Team  Research  Reports  are 
graded  from  0  to  100  on  seven  internal  criteria  (executive  summary,  background,  technical  discussion, 
ethical  factors,  recommended  actions,  organization  and  references)  that  are  also  combined  to  form  the 
overall  0-100  score.  Student  data  are  available  from  the  class  roster  (name,  major,  gender)  and  from 
recorded  performance  scores  on  the  two  individual  writing  assignments,  the  midterm  and  final  exams, 
and  the  Report  components. 

Study  Scope  and  Objectives.  In  the  present  study  we  used  only  183EW  data  that  is  historical  (i.e.,  past 
quarters  only)  and  anonymous  (i.e.,  no  student  names  connected)  to  examine  the  existing  correlations 
between  team  member  characteristics  and  the  performance  of  the  total  team  in  performing  this  typical 
engineering  task. 

Our  analysis  focused  on  the  period  after  Engr  183  became  a  requirement  for  the  entire  School  of 
Engineering  and  Applied  Science  (this  increased  the  diversity  of  student  majors)  and  before  it  also 
became  an  Engineering  Writing  course  (the  additional  TA  attention  both  raised  the  level  and  narrowed  the 
range  of  Team  Report  scores).  Consequently,  the  team  candidate  list  comprised  student  from  five  prior 
quarters  (from  Spring  ’07  through  Fall  ’08);  As  shown  in  the  table  below,  it  involved  approximately  700 
students  and  exactly  130  teams. 


Quarter 

Students 

Teams 

1 .  Fall  '08 

172 

34 

2.  Spring  '08 

145 

27 

3.  Winter  '08 

129 

29 

4.  Fall  '07 

107 

22 

5.  Spring  '07 

138 

24 

Total 

691 

130 

A  primary  objective  was  to  allow  further  testing  of  the  ATCS  using  optimization  criteria  based  on 
knowledge  derived  from  an  existing  “personnel”  data  base.  A  second  objective  was  to  use  the  empirical 
correlations  for  further  refinement  of  the  Team  Performance  Framework,  helping  shed  light,  for  example, 
on  the  relative  value  of  team  diversity  vs.  team  homogeneity  (based  on  such  measures  as  individual 
performance,  major,  general  ethnicity,  etc.)  for  team  performance  (based  on  the  overall  performance 
score  or  constituent  scores).  A  third  and  different  objective  was  to  potentially  benefit  the  Engr  1 83  course 
itself  by  helping  the  instructors  to  guide  students  into  better  performing  teams  and  to  provide  extra 
assistance  in  areas  where  already-formed  teams  are  likely  to  need  it.  The  results  may  also  benefit  the 
UCLA  SEAS  as  a  whole  by  providing  more  information  about  the  factors  that  contribute  to  the  success  of 
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engineering  teams,  and  UCLA  or  other  academic  organizations  may  also  in  the  future  be  able  to  make 
use  of  the  Perceptronics  Solutions  automated  team  composition  system  that  results  in  part  from  the 
research  results. 

Data  Anonymity  Scheme.  The  historical  data  were  formatted  by  the  Engr  183  course  instructors  and 
delivered  under  a  subcontract  to  UCLA  SEAS  from  Perceptronics  Solutions.  Figure  2  shows  the 
preliminary  data  format  and  the  anonymity  scheme.  The  student  names  in  the  Engr  183  Course  data 
were  converted  to  numbers  by  the  course  instructors  as  the  first  step  of  the  study,  and  the  analysis  of 
individual  characteristics  to  team  performance  was  based  entirely  on  these  numbers  and  did  not  involve 
student  names  or  other  personal  identification.  There  was  be  no  way  for  an  outsider  (including 
Perceptronics  Solutions  staff)  to  trace  these  data  back  to  a  particular  student  because  the  quarter  in 
which  it  was  obtained,  the  specific  team  topic  involved,  the  student  UCLA  identification  number,  and  any 
other  identifying  material  was  either  redacted,  randomized  or  anonymized.  The  existing  class  data 
containing  student  names  and  performance  records  remained  in  the  current  data  files  accessible  only  to 
the  course  instructors  and  could  not  be  further  released  --  in  accord  with  all  privacy  and  security  concerns 
and/or  regulations.  This  scheme  was  approved  by  the  UCLA  Registrars  Office  and  also  by  the  UCLA 
Internal  Review  Board  (IRB).  To  emphasize:  The  data  used  was  only  archival  and  did  not  involve  or 
affect  any  students  previously  or  currently  enrolled  in  the  class,  and  the  data  was  anonymous,  student 
names  having  been  removed  in  the  initial  processing  step. 


Quarter 

Redacted 

Team  # 
Randomized 

Student  ID 
Anonymized 

Student  # 
Redacted 

Student  Final 
Grade  Redacted 

derrPEJata  Individual  Performance  Team  Performance 


Team 

Essay 

Essay 

Midterm 

Final 

Final 

Exec 
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Figure  2  UCLA  Study  Sample  Data  and  Anonymity  Scheme 

Data  Analysis.  We  exercised  the  data  base  both  with  some  simple  "warm-up"  analyses  and  a  number  of 
more  complex  analyses  of  the  correlations  among  individual  student  characteristics,  team  diversity 
measures  and  team  performance  scores. 
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Our  independent  variables  were  based  on  data  covering  individual  student  performance  (two  essays,  a 
midterm  exam,  a  2-part  final  exam),  student  participation  (class  attendance  and  observed  team 
contribution)  as  well  as  student  characteristics  (engineering  major,  student  gender  as  determined  by  first 
name  and  roster  picture,  and  an  informal  designation  of  student  ethnicity  as  determined  primarily  by  his  or 
her  name).  Coding  for  ethnicity  was  as  follows: 

1.  ASI;  Asian,  includes  Chinese,  Korean,  Japanese,  etc.  names 

2.  AME;  American/European,  includes  all  English,  Irish,  Russian,  etc.  names 

3.  HIS;  Hispanic,  includes  Mexican  and  Spanish  names 

4.  INP;  Indian  and  Pakistani  names 

5.  MDE;  Middle  Eastern,  includes  Arabic,  Israeli  and  Armenian  names 

6.  AFR;  African,  includes  only  African  names,  African-Americans  are  included  in  AME 

Coding  for  academic  major  was  as  follows. 


Abbreviation  Aggregated 

CHM 

CIV 

COM 

ELE 

MEC 

Other  AER 

BIO 
CSE 
MAT 
DBLE 


Program  Name 

Chemical  Engineering 
Civil  Engineering 
Computer  Science 
Electrical  Engineering 
Mechanical  Engineering 
Aerospace  Engineering 
Bioengineering 

Computer  Science  and  Engineering 
Materials  Science 
Double  Major 


In  the  case  of  ethnicity  and  academic  major,  low  frequency  categories  were  sometimes  collapsed  to  avoid 
empty  cells  in  factorial  analyses.  With  respect  to  ethnicity,  Hispanics,  Indians  and  Pakistanis,  Middle 
Easterners,  and  Africans  were  sometimes  combined  into  the  single  “Other”  category. 

Dependent  variables  were  the  Team  Research  Report  Scores,  including  the  seven  component  scores 
and  the  overall  Report  score. 

The  following  presents  those  results  from  our  analyses  that  pertain  most  directly  to  the  identification  of 
relevant  measures  that  might  be  useful  as  inputs  to  the  Automated  Team  Composition  System.  We 
exercise  some  caution  in  the  interpreting  these  results  for  a  variety  of  reasons;  these  include:  the  relative 
homogeneity  of  the  student  population,  ceiling  effects  on  grades,  the  non-random  selection  of  team 
members,  and  the  large  number  of  possible  effects  examined,  leaving  wide  scope  for  chance.  The 
analyses  are  accordingly  exploratory:  We  report  significant  effects  (p  <  0.05),  marginally  significant  effects 
(.05  <  p  <_0.10)  and  non-significant  trends  (.10  <  p  <  0.1 5+). 

3.  Analytical  Results 

Data  Base  Characteristics.  The  data  base  of  approximately  700  students  contained  80%  males  and 
20%  females.  There  were  significantly  more  Asians  (52%)  than  American/Europeans  (28%)  or  Other 
(20%).  Using  the  latter  three  categories  in  a  Chi-squared  test,  this  pattern  held  without  significant 
variation  across  quarters.  Students  were  not  spread  evenly  across  academic  majors.  Overall,  23%  of 
students  were  in  Electrical  Engineering;  17%  in  Mechanical  Engineering;  13%  in  Computer  Science;  13% 
in  Civil  Engineering;  and  11%  in  Chemical  Engineering.  All  other  fields,  totaling  23%,  had  less  than  10% 
of  the  students. 
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Figure  3  (left)  shows  the  number  of  teams  with  specified  number  of  male  team  members.  There  were  no 
teams  with  zero  male  members.  Figure  3  (right)  shows  the  number  of  teams  with  specified  number  of 
female  team  members.  There  were  a  large  number  of  teams  with  no  females;  the  mean  was  one  female. 


_  _  Male_sum  Fem_Sum 

Figure  3  Distribution  of  male  team  members  (left)  and  female  team  members  (right). 

Team  Diversity.  A  direct  way  to  study  the  effect  of  diversity  is  by  a  diversity  measure.  We  did  this  by 
means  of  Shannon  entropy.  It  is  a  widely  used  measure  of  dispersion  among  nominal  values,  which  plays 
the  same  role  as  variance  does  for  dispersion  of  numerical  values.  The  formula  for  Shannon’s  index  of 
diversity  is: 

D  =  -  Z  pi  ln(pO 

where  pi  is  the  proportion  of  the  population  represented  by  subgroup  i.  Diversity  is  at  a  maximum  when 
each  of  n  subgroups  represents  1/n  of  the  population;  the  maximum  in  that  case  is  ln(n).  Diversity  is  at  a 
minimum,  equal  to  0,  when  the  entire  population  is  concentrated  in  one  subgroup. 

There  was  a  good  amount  of  variation  across  teams  on  ethnicity  and  academic  major  as  exemplified  by 
Figure  4  showing  the  distribution  of  diversity  measure  for  ethnicity  and  majors.. 


Figure  4  Distribution  of  Teams  by  Diversity  Measure  for  Ethnicity  (left)  and  Academic  Major  (right) 
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Individual  Grade  Scores.  We  found  that  Individual  grades  of  different  types  are  correlated  with  one 
another  across  students,  and  moreover,  these  correlations  yield  the  two-dimensional  Euclidean  distance 
solution  shown  in  Figure  5. 


-2-10  1  2 


Dimension  1 

Figure  5  Multi-dimensional  scaling  in  which  greater  distances  represent  less  correlation 
among  different  kinds  of  individual  student  scores. 

The  two-dimensional  solution  provides  a  very  good  fit  to  the  data,  as  evidenced  by  low  stress  and  high 
dispersion  accounted  for.  Moreover,  the  different  grades  appear  to  cluster  in  a  very  natural  way:  With 
respect  to  dimension  2,  participation  is  at  one  extreme  and  products  (essays  and  exams)  at  the  other. 

With  respect  to  dimension  1 ,  essay  products  are  on  the  left  and  test  products  on  the  right.  This  intuition  is 
confirmed  by  hierarchical  cluster  analysis,  as  shown  below.  At  the  level  of  three  clusters,  we  have  distinct 
clusters  for:  (1)  participation,  (2)  the  three  tests,  and  (3)  the  two  essays. 

Influence  of  Individual  and  Team  Variables  on  Team  Performance.  Table  1  on  the  following  page  is  a 
summary  of  the  significant,  marginally  significant  and  trend  effects  determined  from  our  various  data 
analyses.  The  effects  themselves  are  a  combination  of  the  contribution  to  team  performance  made  by 
single  individuals  with  a  particular  attribute  (e.g.,  Computer  major)  and  the  contribution  made  by  having 
increasing  numbers  of  individuals  with  that  attribute  (e.g.  several  members  with  Computer  majors).  The 
goal  is  to  give  an  idea  of  how  the  candidates’  attributes  affect  the  performance  of  the  teams  of  which  they 
are  a  member.  Our  review  of  this  summary  suggests  a  number  of  relevant  findings. 

•  Gender.  There  is  a  clear  advantage  to  having  female  team  members,  who  contribute  positively 
almost  across  the  board 

•  Major.  Some  majors  have  a  balance  of  positive  and  negative  effects,  but  it  appears  that  Material 
Science  and  Mechanical  Engineering  majors  are  detrimental  to  team  success,  while  Aerospace 
and  Bioengineering  majors  are  beneficial  over  a  wide  range  of  Report  components. 

•  Ethnicity.  There  is  a  tendency  for  Asian  students  to  add  to  team  performance  in  the  Technical 
area  and  the  Executive  Summary  but  not  in  the  Ethics  or  Organization  area,  while 
American/European  students  add  in  Ethics  as  well  as  Technical  but  not  in  the  Executive 
Summary.  This  suggests  a  cultural  difference  in  the  approach  to  or  the  importance  given  to  the 
Ethical  considerations.  ‘Other’  ethnicities  contribute  positively  in  several  areas,  but  not  to  the 
Executive  Summary;  and  there  was  also  a  trend  for  Hispanics  to  positively  influence  Background 
scores. 
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•  Individual  Scores.  Good  performance  on  tests,  whether  separate  or  averaged,  has  its  main 
effect  on  the  Technical  section,  which  seems  understandable,  but  the  average  essay  and  test 
scores  have  a  negative  effect  on  recommended  actions,  which  suggests  that  students  who  are 
good  at  working  with  the  text  and  lecture  material  may  not  also  be  good  at  solving  problems  not 
previously  encountered.  Participation  also  has  a  positive  effect  on  the  Technical  section.. 

Table  1  Summary  of  the  Data  Analyses 


Team  Member 
Variables 

Team  Research  Report  Variables 

Exec 

Back 

Technical 

Ethics 

Actions 

Org 

Ref 

Report 

Gender 

Male 

[Neg] 

Female  (Pos)  (Pos)  (Pos)  (Pos)  (Pos) 

Major 

CHM 

(Neg) 

CIV 

COM 

<Neg> 

[Pos] 

<Neg> 

(Neg) 

ELE 

(Pos) 

(Pos) 

MAT  [Neg]  <Neg>  (Neg) 

MEC  [Neg]  [Neg]  [Neg]  [Neg] 

AER  (Pos)  [Pos] 

BIO  [Pos]  [Pos]  [Pos]  [Pos] 

CSE 

Ethnicity 

ASI  (Pos) 

[Neg] 

AME  [Neg] 

Other  [Neg]  [Pos]  [Pos] 

Individual 

Scores 

Essay  1 

Essay  2 

Midterm 

<Pos> 

Final  1 

[Pos] 

[Neg] 

Final  2 

Ave  Grade1 

<Pos> 

Participation 

(Pos) 

Team 

Scores 

Mean  Ave 
Grade 

(Pos) 

(Neg) 

[Neg] 

Mean 

Participation 

[Pos] 

Team  Size 

-- 

(Pos) 

[Pos] 

Diversity 

Ethic 

Major 

<significant>,  p  <  0.05;  (marginal),  0.05  >  p  <  0.1 0;  [trend],  p  >  0.10 


•  Team  Scores.  The  mean  of  team  members’  average  essay  and  test  scores  reflects  the 
individual  effects  -  positive  on  the  Technical  section  and  negative  on  Recommended  Actions. 
Likewise  the  mean  Participation  score  has  a  positive  effect  on  the  Technical  section 

•  Team  Size.  Increasing  team  size  had  positive  effects  on  the  Ethics  section  as  well  as  on  the 
Technical  section. 

•  Diversity.  Neither  Ethnic  nor  Major  diversity  had  any  significant,  marginal  or  trend  effect  on  any 
of  the  team  performance  measures;  i.e.,  teams  with  high  diversity  performed  as  well  as  teams 
with  lower  diversity. 

Discussion.  The  extensive  data  analysis  summarized  in  the  above  table  has  led  us  to  some  (very) 
preliminary  conclusions  regarding  the  three  main  questions  addressed  in  this  study,  namely: 
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•  What  do  we  learn  about  team  performance  in  general? 

•  How  would  we  use  the  results  to  form  optimal  teams  using  the  ACTS  tool? 

•  How  might  we  use  the  results  to  improve  instruction  in  this  specific  course? 

We  provide  some  initial  answers  in  the  following. 

•  General  Team  Performance.  Our  results  from  this  study  suggest  that: 

o  Team  members  in  an  academic  setting  will  self-select  (perhaps  with  some  degree  of  help)  for 
significant  levels  of  diversity  both  in  ethic  and  technological  dimensions. 

o  Diversity  itself  may  not  improve  or  degrade  performance,  but  it  may  have  other  benefits  in  the 
overall  task  context  and  with  respect  to  other,  broader  objectives.  In  the  current  analysis, 
diversity  per  se  is  seen  to  be  less  important  than  contributions  of  specific  groups  to  specific 
team  competencies. 

o  Females  can  be  a  general  asset  to  team  performance  in  a  technical  context,  and  may  exert 
one  of  the  strongest  effects. 

o  Members  from  different  ethnic  backgrounds  can  contribute  differently  to  specific  task 
elements.  Their  contributions  may  reflect  their  cultural  backgrounds. 

o  Individual  task  skills,  reflected  in  this  case  by  academic  test  scores,  may  affect  only  limited 
team  task  areas  -  in  this  case  only  the  Technical  and  Recommended  Actions  portions  of  the 
Report. 

o  Teams  composed  of  members  with  high  individual  performances  on  constrained  tasks  such 
as  test  taking  may  do  worse  on  team  tasks  requiring  more  creativity  or  decision  making  -  in 
this  case  the  Recommended  Actions  portion  of  the  report. 

•  ATCS  Team  Formation.  Establishing  optimization  criteria  and  constraints  for  the  ACTS  tool 

might  make  use  of  present  results,  including  the  following  areas. 

o  Possible  team  role  weightings.  A  Technical  Lead  might  be  weighted  higher  on  average  test 
scores;  other  team  roles  would  likely  have  different  weightings  for  Essay  scores,  Exam 
scores  and  Participation  score. 

o  Possible  role  constraints.  Constraints  on  individual  roles  might  take  into  account  the  positive 
influences  of  Females,  Bioengineer  and  Aerospace  Engineering  majors,  as  well  as  the 
negative  effects  of  other  major 

o  Possible  team  constraints.  These  might  consider  both  the  results  of  the  correlation  analysis 
and  the  other  objectives  of  the  course,  which  include  having  students  work  on  teams  of 
significant  diversity. 

•  Use  of  Results  for  Improved  Instruction.  The  instructors  of  Engr  1 83  as  well  as  of  other 

Engineering  course  might  make  use  of  the  results  in  the  following  ways: 

o  Recognize  the  student  distributions  in  terms  of  Ethnicity,  Gender  and  Majors. 

o  Identify  females  as  a  particularly  valuable  team  asset,  which  perhaps  should  be  rationed. 

o  Enforce  the  modal  team  size  of  5  as  optimal  -  not  too  small,  not  too  large  -  even  though 
some  performance  improves  with  increasing  team  size. 

o  Observe  that  the  objective  of  promoting  specific  types  of  team  diversity  seems  to  be  working. 

o  Recognize  that  individual  performance  has  three  clear  and  somewhat  separate  dimensions  - 
Essays,  Examinations  and  Participation. 

o  Recognize  that  Asian  students  in  general  may  be  having  problem  with  the  Ethics  area 

possibly  due  to  a  significant  number  of  recent  arrivals  to  the  US,  English  language  problems, 
or  cultural  differences,  and  might  require  special  attention  to  ensure  that  the  course 
objectives  in  this  are  understood  and  integrated  into  the  Team  Research  Report 
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o  And  finally,  recognize  that  some  Majors  seem  to  be  having  problems  with  the  more  societal 
aspects  of  the  course  -  and  this  might  warrant  a  closer  look  at  what  characteristics  might  be 
causing  this  and  what  could  be  done  to  counteract  it. 

In  the  following  section,  we  show  how  we  used  the  analytical  results  to  program  the  ATCS  tool  for  the 
rapid  formation  of  optimal  teams  capable  of  producing  a  strong  Team  Research  Report. 

4.  Team  Formation  Using  the  ATCS  Tool 

The  hypothetical  real-world  problem  was  to  rapidly  form  the  best  team  to  produce  a  Team  Research 
Report  having  at  hand;  (1)  the  general  objectives  of  the  course;  (2)  the  700-student  roster  and 
accompanying  data;  (3)  the  results  of  the  present  correlation  analysis;  as  well  as  (4)  the  general  team 
performance  research  literature  and  in  particular  our  preliminary  formulation  of  a  Team  Performance 
Framework. 

Team  and  Roles.  To  address  this  problem  we  used  the  same  set-up  of  the  ATCS  as  we  had  for  the 
Kaimalu  O  Hawaii  (KOH)  demonstration  conducted  in  May  2009  with  the  assistance  of  the  Center  for 
Asymmetric  Warfare  (CAW)  as  part  of  a  Port  of  Hawaii  anti-terrorist  exercise  [9],  In  the  KOH  case  we 
formed  a  virtual  six-person  command  team  to  oversee  the  anti-terrorist  operations;  in  the  present  case  we 
formed  a  virtual  five-person  team  to  conduct  research  and  produce  a  Team  Research  Report  as 
previously  analyzed.  For  the  research  team  we  defined  the  distinct  roles  shown  in  Table  2,  assuming  a 
specified  Lead  person  for  each  of  several  key  Research  Report  components. 

Table  2  Team  Roles  for  the  Engr  183  Research  Report  Mission 


1 

Organization  &  References  Lead  (ORL) 

2 

Executive  Summary  Lead  (ESL) 

3 

Background  &  Technology  Lead  (BTL) 

4 

Ethics  and  Societal  Lead  (BTL) 

5 

Recommended  Actions  Lead  (RAL) 

Team  Candidate  Characteristics  and  ATCS  Model  Factors.  In  the  KOH  case  the  candidates’  Individual 
Characteristics  comprised  their  agency  (local  police,  FBI,  DHS,  etc.),  their  position  or  rank;  their  Mission 
Factors  came  from  self-assessment  of  experience/skills  in  a  number  of  mission-related  technical  areas 
(HAZMAT,  civil  aviation,  rescue  operations,  etc.);  their  Teamwork  Factors  were  assessed  by  a  20- 
question  survey;  and  their  social  Network  Factors  was  assessed  by  self-reported  recent  communications. 

In  the  present  case  the  candidates’  individual  characteristics  comprised  their  engineering  major,  their 
gender  and  their  ethnicity;  their  mission-related  experience/skill  (taskwork)  factors  came  from  their  test 
and  essay  scores;  and  their  teamwork  factors  came  from  their  participation  score.  No  Network  Factors 
were  applied. 

For  the  present  case  we  defined  two  types  of  Mission  Factors,  as  follows;  these  were  based  on  the 
analytical  findings  regarding  the  clustering  of  performance  scores: 

1 .  Research/Exposition  Skills  =  Ave[Essay1  +  Essay2] 

2.  Course  Purpose/Content  Knowledge  =  Ave[Midterm  +  Finall  +  Final2] 

We  defined  a  weighted  multi-attribute  Mission  Score  (MASCm)  as  follows,  and  equal  weighting  was  used 
for  the  initial  trials. 

MASCm  =  wR[Research/Exposition]  +  wP[Purpose/Content] 

We  defined  a  weighted  multi-attribute  Teamwork  Score  (MASCj)  was  in  this  case  simply  as: 

MASCm  =  [Participation] 
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Role  and  Total  Score  Weighting.  Table  3  table  shows  how  the  Mission  and  Teamwork  Factors  were 
weighted  for  the  individual  team  roles  in  our  trial  runs.  The  weights  were  based  in  part  on  the  analytical 
finding  and  in  part  on  our  assumptions  regarding  the  skills  required  for  each  role. 


Table  3  Factor  Weights  by  Role  for  the  Engr  183  Team 


Organization 
&  Refs  Lead 

Executive 
Sum  Lead 

Background 
&  Tech  Lead 

Ethics 

Lead 

Actions 

Lead 

Indiv  Research/Exposition 

0.15 

0.50 

0.30 

0.20 

0.40 

Course  Purpose/Content 

0.35 

0.35 

0.40 

0.60 

0.30 

Teamwork  Skills 

0.50 

0.15 

0.30 

0.20 

0.30 

In  the  ATCS  we  define  a  Total  Score  used  for  team  optimization  by  the  multiagent  system;  in  the  present 
case  this  could  be  expressed  as: 

Total  Score  =  wmMASCm  +  wTMASCT 

In  the  present  case,  we  assumed  that  the  Mission  Factors  were  more  important  than  the  Teamwork 
Factors,  and  so  assigned  weights  of  0.70  and  0.30  to  these  components,  respectively 

Role  and  Team  Constraints.  We  applied  the  role  constraints  shown  in  Table  4  in  accord  with  the 
associated  objectives,  based  primarily  on  the  relationships  between  team  member  characteristics  and 
team  performance  resulting  from  the  prior  data  analysis. 


Table  4  Role  Constraints  Used  in  the  Engr  183  Team  Formation  Trials 


Role  Constraint 

Objective 

BTL  must  be  BIO  Major 

Strengthen  Background  and  Technology  Section 

BTL  must  have  Purpose/Content  >  90.0 

Strengthen  Background  and  Technology  Section 

RAL  must  be  AERO  Major 

Strengthen  Recommended  Actions  Section 

ESL  must  be  Female 

Strengthen  Ethics  and  Societal  Section 

For  the  sake  of  these  trials,  we  assumed  it  would  be  acceptable  to  specify  a  female  for  a  particular  role, 
as  well  as  to  specify  the  total  number  of  females  on  the  team,  since  females  were  clearly  shown  to  a 
valuable  asset  in  the  data  analysis,  and  planned  representation  in  this  arena  is  generally  approved  (even 
on  the  Supreme  Court).  We  made  the  opposite  assumption  in  the  case  of  ethnicity,  i.e.,  we  imposed  no 
role  or  team  constraints  based  on  ethnicity,  reasoning  that  it  would  not  be  acceptable  in  this  type  of 
situation  to  specify  a  particular  ethnicity  for  a  role  or  to  specify  the  number  of  members  of  particular  ethic 
groups  for  the  team,  even  in  the  interests  of  ethnic  diversity  and  with  regard  to  our  analytical  results.  (This 
assumption  might  be  questioned  in  this  and  other  situations). 

With  respect  to  diversity  of  engineering  major,  however,  we  recognized  that  a  major  objective  of  Engr 
1 83,  as  described  in  the  Course  Syllabus,  is  to  give  students  the  experience  of  working  on  a  technically 
diverse  team.  Accordingly,  we  included  in  the  Team  Constraints  shown  in  Table  5  the  requirement  that 
there  be  at  most  one  member  from  a  particular  major. 


Table  5  Team  Constraints  Used  in  the  Engr  183  Team  Formation  Trials 


Team  Constraint 

Objective 

At  least  one  Female 

Strengthen  Total  Team  Performance 

Not  more  than  two  Females 

Maintain  Gender  Diversity 

Not  more  than  one  member  per  Major 

Maintain  Major  Diversity 
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Team  Composition.  We  limited  the  ATCS  compute  time  to  1  minute  on  the  assumption  that  this  would 
be  a  reasonable  and  acceptable  run  duration,  and  because  we  were  interested  in  whether  the  ATCS 
could  find  optimal  teams  in  this  relatively  short  time. 

The  ATCS  was  successful  in  forming  an  optimal  team  within  one  minute  from  the  approximately  700 
member  candidate  pool  and  the  initial  role  and  team  constraints.  Table  6  shows  the  team  makeup  as  well 
as  the  Optimal  Score,  which  is  a  relative  measure  of  optimality.  The  four  Role  Constraints  met  are  shown 
in  bold  type;  and  the  team  also  meets  the  Team  Constraints  of  not  more  than  two  Female  Members  and 
complete  diversity  of  engineering  majors. 


Table  6  Optimal  Team  Composed  from  Full  Candidate  Data  Base 


Team  1  (Optimal  Score=15993) 

ID 

Major 

Gender 

Ethnicity 

Tests 

Essays 

1 

Org  &  Ref  Lead 

581 

CHM 

Male 

ASI 

98.5 

100 

2 

Executive  Summary  Lead 

580 

ELE 

Female 

ASI 

96.0 

100 

3 

Background  &  Tech  Lead 

121 

BIO 

Male 

AME 

90.8 

94.5 

4 

Ethics  Lead 

645 

COM 

Male 

INP 

98.9 

99.5 

5 

Recommended  Actions  Lead 

480 

AER 

Female 

AME 

93.9 

100 

We  also  examined  the  new  case  of  forming  two  teams  simultaneously,  where  the  teams  cannot  share 
members,  and  each  team  should  be  as  close  to  optimum  as  possible.  This  is  a  common  situation  where 
multiple  teams  must  be  formed  to  perform  different  parts  of  an  overall  mission,  and  the  teams  cannot  be 
considered  primary  and  backup  but  must  each  be  as  nearly  equivalent  in  capability  as  possible.  The 
standard  unaided  method  of  picking  one  team  and  then  picking  the  next  team  from  the  remaining 
candidates  does  not  satisfy  the  formal  conditions  of  optimality.  In  this  case,  we  were  able  to  exercise  the 
ATCS  in  a  dual-team  optimal  mode  with  the  following  results.  As  seen  in  Table  7,  the  two  optimal  teams 
meet  all  the  role  and  team  constraints,  and  have  a  common  Optimal  Score  just  slightly  lower  than  the 
single  optimal  team  above. 


Table  7  Two  Optimal  Teams  Composed  from  Full  Candidate  Data  Base 


Team  1  (Optimal  Score=15860) 

ID 

Major 

Gender 

Ethnicity 

Tests 

Essays 

1 

Org  &  Ref  Lead 

645 

COM 

Male 

IDP 

98.6 

99.5 

2 

Executive  Summary  Lead 

144 

CIV 

Female 

AME 

97.2 

93.5 

3 

Background  &  Tech  Lead 

642 

BIO 

Female 

ASI 

97.8 

98.5 

4 

Ethics  Lead 

149 

ELE 

Male 

ASI 

97.8 

98.0 

5 

Recommended  Actions  Lead 

633 

AER 

Male 

AME 

83.5 

96.0 

Team  2  (Optimal  Score=15860) 

ID 

Major 

Gender 

Ethnicity 

Tests 

Essays 

1 

Org  &  Ref  Lead 

250 

MEC 

Male 

ASI 

99.0 

96.5 

2 

Executive  Summary  Lead 

370 

CHM 

Female 

AME 

94.2 

97.0 

3 

Background  &  Tech  Lead 

531 

BIO 

Male 

ASI 

98.6 

90.5 

4 

Ethics  Lead 

580 

ELE 

Female 

ASI 

96.0 

100 

5 

Recommended  Actions  Lead 

38 

AER 

Male 

ASI 

93.5 

97.0 
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Interestingly,  while  we  did  not  impose  any  role  or  team  constraints  based  on  ethnicity,  overall  the 
percentage  of  Asians  (60%),  American-Europeans  (30%)  and  Others  (10%)  in  the  two  selected  teams  is 
quite  well  reflective  of  the  candidate  population  in  which  Asians  are  52%,  American-European  are  28% 
and  Others  are  20%. 

Finally,  we  examined  the  case  of  using  only  half  the  candidate  data  base  for  team  composition,  because 
this  increased  the  number  of  alternative  team  configurations  that  the  ATCS  could  examine  during  the 
same  computational  time.  For  this  trial  the  approximately  700  member  candidate  pool  was  randomly 
divided  into  A  and  B  halves.  Table  8  shows  the  results  in  terms  of  team  member  IDs  in  comparison  to 
two  teams  formed  with  the  full  data  base.  As  seen,  the  half-DB  results  share  some  role  assignments  with 
the  full-DB  runs,  but  not  many.  In  addition,  the  optimality  scores  for  the  half-DB  teams  are  the  same  or 
even  slightly  higher  than  those  for  the  full-DB  case,  which  is  consistent  with  the  ATCS  being  able  to 
process  more  team  options  when  it  is  working  with  a  smaller  candidate  pool.  It  appears  that  when  the 
candidate  pool  is  this  large,  dividing  it  in  half  randomly  does  not  materially  affect  team  composition. 


Table  8  Comparison  of  Teams  from  Full  and  Half  Candidate  Data  Bases 


Team  Role 

Full  Data  Base 

Half  Data  Base  A 

Half  Data  Base  B 

1 

Org  &  Ref  Lead 

581 

581 

250 

645 

581 

618 

2 

Executive  Summary  Lead 

580 

580 

580 

580 

83 

36 

3 

Background  &  Tech  Lead 

642 

121 

642 

642 

20 

20 

4 

Ethics  Lead 

645 

645 

645 

250 

618 

651 

5 

Recommended  Actions  Lead 

633 

480 

633 

633 

480 

480 

Optimal  Score 

16030 

15993 

16023 

16020 

15990 

15984 

5.  Conclusions 

End-to-End  Demonstration.  The  ATCS  team  composition  runs  successfully  concluded  the  UCLA  Engr 
1 83  archival  study  as  a  complete  end-to-end  demonstration  of  the  following  major  team  composition 
processes: 

1 .  Extracting  from  an  existing  personnel  data  base  the  information  about  individual  characteristics 
and  performance  needed  for  automated  team  composition,  as  would  be  the  case  in  many 
envisioned  applications; 

2.  Using  knowledge  about  the  factors  influencing  team  performance  for  a  well  defined  mission  along 
with  knowledge  about  general  mission  objectives  and  team  performance  factors  to  establish  the 
ATCS  model  parameters  and  optimization  criteria,  as  well  as  the  role  and  team  constraints; 

3.  Using  the  programmed  ATCS  to  form  optimal  teams  in  an  acceptable  computational  time  frame 
using  a  substantial  candidate  pool  and  typical  model  parameters  and  constraints. 

The  teams  formed  in  the  UCLA  case  certainly  have  great  face  validity:  They  have  the  right  people  in  the 
right  roles,  as  well  as  a  high  degree  of  diversity.  They  appear  fully  representative  of  teams  that  would  be 
selected  manually,  but  have  the  major  advantages  of  known  optimality  using  all  programmed  criteria,  in 
addition  to  very  rapid  formation  despite  a  large  candidate  pool  -  features  impossible  to  achieve  manually. 

Because  we  used  average  Essay  and  Test  scores  as  Mission  Factors  in  our  team  optimization  criteria, 
the  team  members  selected  have  almost  uniformly  high  scores  for  these  taskwork  variables.  There  was 
some  suggestion  in  the  statistical  analysis  (as  well  as  in  the  literature)  that  having  a  team  in  which  all 
members  have  a  high  individual  performance  level  may  not  yield  the  best  total  team  performance  in  all 
aspects  of  the  mission.  Accordingly,  another  useful  team  constraint  might  be  to  include  some  number 
of  lower  scoring  members  (i.e.,  to  impose  Grade  Diversity),  even  though  this  is  basically  counter-intuitive. 
Likewise,  in  some  situations  the  use  of  ethnicity  data  might  be  acceptable,  and  this  could  add  another 
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dimension  of  optimality  to  the  present  results.  Our  planned  experimental  research  may  help  clarify  this 
point. 

Relationship  to  Future  R&D.  The  ATCS  project  contains  a  number  of  innovative  elements  that  will 
influence  future  research  and  development  in  the  related  research  fields.  These  elements  include: 

•  Structuring  the  immediate  and/or  long  term  mission  of  an  existing  organization  in  terms  of  a 
formal  decision  model  in  order  to  quantitatively  represent  important  characteristics  as  part  of  the 
team  optimization  criteria.  This  is  a  significant  contribution  to  practice  in  both  organizational  and 
team  performance  research  domains  because  it  effectively  combines  methodologies  that  are  not 
usually,  if  ever,  brought  together. 

•  Enabling  the  combination  of  empirical  results  from  team  performance  studies  with  automated 
team  composition  methodology  based  on  multiagent  system  technology,  where  the  first  serves  as 
an  input  to  the  latter.  This  is  a  totally  new  approach  and  will  be  a  major  contribution  to  research 
in  both  areas. 

•  In  the  future,  using  the  results  of  the  automated  team  composition  system  as  a  means  of 
evaluating  both  the  contribution  of  the  various  factors  previously  identified  with  effective  team 
performance  and  as  a  means  of  testing  empirically  whether  teams  formed  on  the  basis  of 
previously  identified  key  factors  actually  perform  better  in  their  processes  and  outcomes. 

In  essence,  we  are  bringing  together  two  disciplines:  one,  team  performance  modeling,  including 
investigation  of  how  teams  evolve;  and  two,  proactive  computer-supported  team  composition  through  the 
use  of  advanced  multiagent  system  technology.  This  is  a  significant  contribution  to  team  performance 
research  and  also  an  incentive  to  further  studies  of  multiagent  system  applications.  In  future  planned 
experimental  work  we  will  also  encourage  research  and  development  by  demonstrating  solutions  to  two 
difficult  problems  in  this  field: 

•  What  key  measures  of  team  performance  can  we  get  hold  of  practically  by  using  available 
instruments,  existing  personnel  data  bases,  acceptable  questionnaires,  etc.? 

•  How  can  we  incorporate  those  measures  and  instruments  into  computer-assisted  team 
composition  systems  that  are  both  useful  and  usable? 

•  How  do  we  address  human  and  social  issues  in  the  area  of  computer-assisted  team  composition 
and  improvement?  Problems  can  arise  both  with  individuals  who  want  to  protect  their  privacy 
(e.g.  their  address  book,  their  qualifications,  their  associations,  etc.)  and  with  the  “IT  mafia”  who 
are  afraid  of  compromising  “their”  command  and  control  network. 

We  believe  that  our  solutions,  which  will  be  based  on  continued  comprehensive  literature  research, 
thorough  analyses,  and  empirical  studies  involving  computer-assisted  methodology,  will  be  a  major 
resource  for  future  researchers. 
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